feat(import): add script tool for multiple hbase snapshot imports) by tianlei2 · Pull Request #4606 · googleapis/java-bigtable-hbase

tianlei2 · 2026-04-30T17:44:47Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

b/429250716

This is the second PR with the shell tool script. This a child of #4600, so the only file that needs to be reviewed is the script itself (bigtable-dataflow-parent/bigtable-beam-import/run-snapshot-import.sh). If this looks ok, I will merge this with the parent PR.

The script is originally from https://docs.google.com/document/d/1T7BQF-AYY8xGbdbhwkV7zPA11A_Fh5SRjA_7nkzeTos/edit?tab=t.twzz7rrh6jz7#heading=h.m7p3xy7m76u2

vermas2012 · 2026-06-15T15:40:08Z

+    --serviceAccount=${SERVICE_ACCOUNT} \
+    --usePublicIps=false \
+    --enableSnappy=true \
+    --skipRestoreStep=${SKIP_RESTORE} \


This is good, but how are we passing the restorePath?

I feel like we should have a custom restore path and the script (idempoent by adding timestamp etc) and use it for restore the path and pass it as the restorepath in every job.

Also, with this model, who cleans up the restore path? is there a way to trigger a cleanup at the end of the script? or we have a tool that can be used? We can also say its a manual step. but then this script should output something to the tune of "the snapshot was imported, please cleanup $RESTORE_PATH once validation succeeds."

Added some output for manual cleanups and passing restore path to every job.

vermas2012 · 2026-06-15T15:43:43Z

+
+Example for manual parallel execution:
+```bash
+./run-snapshot-import.sh 0 3 &  # Run this first!


In this model, how does the shard1. know htat it has to wait for restore to finish from shard 0? won't it run and fail becuase restore path is empty?

updating this to run a restore-only command first

…twork argument handling in snapshot import script

tianlei2 · 2026-06-23T17:31:14Z

 		      <exclude>META-INF/*.RSA</exclude>
          <exclude>META-INF/**/pom.properties</exclude>
          <exclude>META-INF/**/pom.xml</exclude>
-          <exclude>META-INF/MANIFEST.MF</exclude>


Removing the MANIFEST.MF exclusion preserves the Multi-Release: true header required to load JDK 21-compatible dependency classes and prevent startup crashes, while remaining fully backward-compatible and safe for JDK 8.

mutianf · 2026-06-23T17:50:46Z

@@ -0,0 +1,188 @@
+#!/bin/bash


nit, maybe we put this file under a bin/ or scripts/ directory?

mutianf · 2026-06-23T17:53:33Z

+# ------------------------------------------------------------------------------
+# Usage
+# ------------------------------------------------------------------------------
+# Usage: ./run-snapshot-import.sh <start_shard> <end_shard>


Both start and end shard are inclusive, right? Probably good to add a note about it?

Also what is a shard?

mutianf · 2026-06-23T17:55:40Z

+END_SHARD=$2
+
+# Configurations
+JAR_PATH="target/bigtable-beam-import-2.18.2-SNAPSHOT-shaded.jar"


Nit: we shouldn't have the jar using a fixed snapshot version. Also, we should probably build the jar first?

mutianf · 2026-06-23T18:00:22Z

+# If running the launcher on a newer, unsupported JDK version (e.g. JDK 25+),
+# you can set JVM_OPTS to "-Dnet.bytebuddy.experimental=true" to bypass
+# ByteBuddy class generation crashes during pipeline construction.
+JVM_OPTS=""


But will this override the settings described in the comment?

mutianf · 2026-06-23T18:37:36Z

+      --restorePath=${RESTORE_DIR} \
+      --jobName="restore-job" \
+      "${NETWORK_ARGS[@]}"
+    echo "✅ Restore completed."


Does the java job has awaitPipelineFinish in the end? Cause otherwise it'll just submit the job and we need to track progress on the dataflow dashboard.

tianlei2 requested a review from a team as a code owner April 30, 2026 17:44

product-auto-label Bot added size: xl Pull request size is extra large. api: bigtable Issues related to the googleapis/java-bigtable-hbase API. labels Apr 30, 2026

tianlei2 changed the title ~~feat(add script tool for multiple hbase snapshot imports)~~ feat(import): add script tool for multiple hbase snapshot imports) Apr 30, 2026

tianlei2 self-assigned this Apr 30, 2026

tianlei2 requested a review from vermas2012 April 30, 2026 17:47

tianlei2 force-pushed the add-snapshot-import-script branch 7 times, most recently from 3086725 to 958c187 Compare April 30, 2026 18:16

tianlei2 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 4, 2026

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label May 4, 2026

tianlei2 force-pushed the add-snapshot-import-script branch 5 times, most recently from 40e4a02 to af3976d Compare May 6, 2026 16:46

vermas2012 reviewed May 6, 2026

View reviewed changes

Comment thread bigtable-dataflow-parent/bigtable-beam-import/run-snapshot-import.sh Outdated

Comment thread bigtable-dataflow-parent/bigtable-beam-import/run-snapshot-import.sh Outdated

tianlei2 force-pushed the add-snapshot-import-script branch from af3976d to 1c83a18 Compare May 13, 2026 15:30

vermas2012 reviewed Jun 15, 2026

View reviewed changes

tianlei2 force-pushed the add-snapshot-import-script branch 2 times, most recently from 9622477 to f9ce742 Compare June 22, 2026 14:31

product-auto-label Bot added size: m Pull request size is medium. and removed size: xl Pull request size is extra large. labels Jun 22, 2026

tianlei2 force-pushed the add-snapshot-import-script branch 3 times, most recently from e2e88b7 to 3b1044b Compare June 22, 2026 15:27

product-auto-label Bot added size: xl Pull request size is extra large. and removed size: m Pull request size is medium. labels Jun 22, 2026

tianlei2 added 2 commits June 23, 2026 17:27

feat(import): add snapshot import helper script and usage documentation

acd4109

Fix JDK 21+ compatibility, restore path directory constraints, and ne…

1e2551c

…twork argument handling in snapshot import script

tianlei2 force-pushed the add-snapshot-import-script branch from 70ae9d4 to 1e2551c Compare June 23, 2026 17:27

product-auto-label Bot added size: l Pull request size is large. and removed size: xl Pull request size is extra large. labels Jun 23, 2026

tianlei2 commented Jun 23, 2026

View reviewed changes

Update SNAPSHOT_IMPORT_USAGE.md troubleshooting section for JVM_OPTS

38715e3

mutianf reviewed Jun 23, 2026

View reviewed changes

Conversation

tianlei2 commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tianlei2 commented Apr 30, 2026 •

edited

Loading